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ABSTRACT 

From a cognitive perspective, racial bias is evident 
when raters weigh job-relevant information differentially as a 
function of ratee race. The results of studies that have examined 
this issue have been conflicting. Meta-analytic procedures were used 
to provide more definitive conclusions as to whether supervisor 
ratings are more strongly related to objective indices of performance 
for black than for white employees. The results from 18 studies 
indicated that the relationship between objective and subjective 
measures varied as a function of ratce race and the type of objective 
criteria used. The ratings of blacks were more related to objective 
indices than were the ratings of whites. Differences in correlations 
by race were found for performance but not for objective job 
knowledge measures. More attention needs to be focused on uncovering 
systematic job relevant and irrelevant variance that is related to 
both subjective and objective indices of performance. (Author/NRB) 
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Abstract 

From a cognitive perspective f racial bias is evident when raters 
weigh job relevant information differentially ass a function of 
ratee race. The results of studies that have examined this issue 
have been conflicting. The present research used meta-analytic 
procedures to provide more definitive conclusions as to whether 
supervisor ratings are more strongly related to objective indices 
of performance for black than for white employees. The results 
from eighteen studies indicated that the relationship between 
objective and subjective measures varied as a function of ratee 
race and the type of objective criteria used. Ratings of blacks 
were more related to objective indices (r=.27) than for whites 
<r».13). Moreover, differences in correlations by race were found 
for performance but not for objective job knowledge measures. 
Implications for the existence of positivity bias in ratings were 
discussed and research directions suggested. 
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The Relationship Among Criteria as a Function of Subgroup 

Membership j An Integrative Review 
For the last twenty years, much research attention has 
focused on the potential for racial bias in personnel decisions. 
Although researchers have acknowledged that important personnel 
issues such as test fairness cannot be addressed without an 
unbiased criterion (Cleary, 1968; Guion, 1966), personnel research 
continues to concentrate on predictor rather than criterion- 
related issues of bias {Burke, 1984). 

xnvestigcbions that have examined the potential for criterion 
bias have largely be*?n simplistic and atheoretical (Feldman, 
1981). The literature is replete with examples of studies that 
compare subgroup mean differences and provide post hoc 
explanations for the presence or absence of bias. However, 
differences in subgroup means and variances are not necessarily 
evidence of bias, nor does the lack of differences* indicate a lack 
of bias. Consequently, this approach to bias has resulted in a 
diverse literature and few substantive conclusions. 

A sew direction for increasing our understanding of the 
nature and covariates of bias is a cognitive, information 
processing approach. Landy and Farr (1980) have provided a basis 
for this approach by defining bias *s the application of different 
mental processes regarding ratees as a function of their subgroup 
membership. This definition implies that raters may selectively 
attend to different behaviors of the ratee, differentially use or 
discriminate among performance dimensions, and/or weight various 
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job relevant and job irrelevant information differentially as a 
function of ratee race. 

Although attempts to directly study these underlying 
processes are rare, a few studies have examined the extent to 
which job relevant information is differentially related to 
ratings as a function of ratee race. Operationally, these studies 
have investigated the extent to which objective performance 
indices are related to subjective ratings of performance. Bass and 
Turner <1973), for example, found that the number of errors made 
by part-time tellers was more related to ratings of "quality of 
work" for black than for white ratees. Cascio and Valenzi (1978) 
used eight objective measures of police performance as predictors 
of a summated rating. The results indicated that identical 
predictor sets validly forecasted supervisory ratings for black 
and white officers and that objective data were related to ratings 
similarly for black and white officers. Flaugher and Norris (1969) 
found that ratings of job knowledge given by white supervisors 
were more highly correlated with actual job knowledge test scores 
for black than for white subordinates. Campbell, Crooks, Mahoney fi 
Rock (1973) also found a stronger relationship between job 
knowledge test scores and ratings of job knowledge for black than 
white medical technicians bub found no difference in this 
relationship for cartographic technicians. 

The above studies have demonstrated that a significant amount 
of the rating variance is related to one or more "objective" 
measures of performance. Nevertheless, the conclusions from these 
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studies are conflicting as to whether ratings are more strongly 
related to objective indices of performance for black than for 
white ratees. 

'^he present study is an attempt to provide more definitive 
conclusions as to the extent to which objective measures of 
performance are differentially related to ratings as a function of 
ratee race. Meta-analysis was used to test the hypothesis posited 
by Cascio and Valenzi (1978) that supervisory ratings are sore 
strongly related to objective indices of performance for black 
than for white employees. In addition, the present study explored 
the possibility that the relationship between objective 
performance and subjective ratings is moderated by the type of 
objective criteria (actual performance or job knowledge indices) 
employed (Ford, Kraiger, & Scheohtman, 1986). 

Method 

An attempt was made to locate and cumulate the results of all 
published and unpublished studies reporting a correlation between 
at least one objective index and an overall rating of performance 
effectiveness for the same sample of black and white employees. A 
total of 18 sample* were located for the analysis and are 
presented in the appendix. 
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Analysis 

The weighted mean correlation (r) of the relationship between 
the objective data and subjective ratings was cumulated by ratee 
race. The variance (a r 2 > across studies, sampling error (a 2 ), and 
the population variance { a ) were computed using procedures 
developed by Hunter, Schmidt and Jackson (1982). The estimated 
standard error ( 0e ) wa s used to establish 95% confidence intervals 
around the appropriate r to test the hypothesis that r = 0 in the 
population. 

The 18 samples in the analysis incluaed 15 different 
objective measures of performance. Using procedures similar to 
Ford et al. (1985) f these criteria were reliably categorized into 
the following two categories: performance indices (N=9) (e.g., 
number of arrests, accidents, attendance) and job knowledge (N-9) 
(e.g., objectively scored written examinations) criteria. Ratings 
of overall effectiveness were used in the analysis when specific 
ratings (e.g., job knowledge, quality of work) were not available. 
The raters in the studies were predominately white. 

Differences in correlation related to criterion type were 
examined by classifying the studies into the appropriate subsample 
and recomputing subsample r's, 0 2 ' B and 95* confidence intervals. A 
moderator is evident when the average correlation varies from 
subset to subset and the corrected variance is lower in the 
subsets than for the data as a whole (Hunter et al., 1982). 
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Results 

The results of the meta-analysis are presented in Table 1. 
The table contains cumulated sample sizes for the black and white 
samples, mean correlations, variance estimates and 95% confidence 
intervals for the 18 studies in the overall analysis and for the 
data as divided by criterion type. 



Insert Table 1 about here 



The best estimate of the population correlation is the 
weighted mean correlation between the objective index and the 
subjective rating of performance. For the overall analysis, the 
weighted mean correlation for the 1337 black ratees in the sample 
was .27. For the 3644 white ratees, the mean correlation was ,13. 
The 95* confidence interval for the black sample rejects the null 
hypothesis that r«0 in the population (.06 < p < .48) while the 
interval includes zero for the white sample (-.005 < P < .26). 

The moderator analyses presented in Table 1 indicate that for 
the performance measures, the mean correlation for black ratees 
(r«.23) varied from the mean correlation for white ratees (r*.06). 
The corrected variance was also lower in the subsets than for the 
data overall . For the job knowledge criteria, the mean 
correlations for the black (r=.31) and white (r=.30) ratees were 
quite similar. 
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Discussion 

This meta-analysis integrates and extends previous research 
(e.g., Bass and Turner, 1973; Cascio and Valenzi, 1978) on ^he 
relationship among criterion measures. The results support the 
hypothesis that ratings are more related to objective data for 
black than for white ratees. The relationship between objective 
and subjective measures of performance was also a function of the 
type of objective criteria used. Ratings of blacks were more 
strongly related to objective measures of job performance, while 
no evidence was provided to support the conclusions of Flaugher 
and Norris (1969) regarding the relationship among job knowledge 
indices, supervisory ratings and ratee race. It is interesting to 
note that the correlation between job knowledge scores and ratings 
of effectiveness is higher than for objective performance indices 
and ratings. This occurs despite the fact that for some studies 
multiple objective performance indices were used to predict 
subjective ratings while only one index of job knowledge was 
correlated with a rating of performance. These results support 
Hunter's (1983) claim that job knowledge is a critical variable 
considered by raterB when making performance evaluations. 

One possible explanation for the present results is that 
given the pressures to be "unbiased" in their evaluations, white 
supervisors may rely on more objective, documented evidence when 
rating black ratees (Cascio & Valenzi, 1973). A second, more 
theoretical explanation is that white supervisors may be more 
psychologically distant from black than from white ratees, thereby 
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reducing the possibility for using non-ob jecti\e compensatory 
factors (i.e., "at least he/she tries hard") when evaluating black 
ratees (Bass and Turner , 1973; Kraiger, 1981) . The 
counterintuitive premise that bias may result from a tendency by 
raters to inflate the ratings of the majority group rather than 
deflate the ratings of the minority group has been labeled 
oositivity bias (Pettigrew, 1979). With positivity bias, the 
minority group that received the more "objective" rating is in 
fact being unfairly evaluated in comparison to the majority group,. 

The existence of a positivity bias has important implications 
for research as it suggesrbs the need to further examine the 
information used and the causal explanations made by raters when 
evaluating ratees. For example , raters could be making positive 
internal attributions ("he/she tries hard") which may compensate 
for poorer actual performance for the one subgroup that is more 
similar to the rater. One technique for exploring this possibility 
is having raters provide verbal protocols (Payne, 1976) and having 
the results coded for type of reasons given for the evaluation of 
performance (ability, effort, task difficulty, luck). The 
responses could also be coded for the frequency of behavioral 
examples or actual job performance data (e.g., attendance) used 
while evaluating individuals of different races. 

Regardless of the explanation, the results indicate that the 
common notion that ratings are "biased" against blacks may not be 
as appropriate as the notion that ratings are biased for whites. 
In other words, differences in ratings as a function of ratee race 
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may partially result from the inflation of the majority group's 
ratings beyond what their performance dictates. 

The results of the present meta-analysis are not meant to 
imply thfct raters use objective information in the same manner as 
indicated by the weighted mean correlations (Cascio and Valenzi? 
1978). The important notion is that more attention needs to be 
focused on uncovering systematic job relevant and irrelevant 
variance that is related to both subjective and objective indices 
of performance. Only with a greater understanding of these 
criterion issues can we hope to improve our understanding of 
predictor-criterion relationships and their linkage to racial 
bias. 
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A Meta-Analysis of the Relationship of Objective and Subjective 
Performance Measures by Ratee Race and Criterion Type 
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No. 

Analysis Studies Subgroup N r 



Overall 



Performance 
Indices 



18 



Job Knowledge 9 
Indices 



Black 1337 .270 .0410 .0116 



White 3644 .130 .0274 .0048 



Black 706 .232 .0158 .0014 



White 2615 .064 .0179 .0034 



Black 631 .312 .0658 .0116 



White 1029 .296 .012797 .0072. 



(table continued) 
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Table 1 (cont.) 

A Meta-Analvsis of the Relationship of Objective and Subjective 
Performance Mea sures by Ratee Race and Criterion Type 

2 2 2 Confidence 

Analysis Subgroup a m oVo' Intervals 

e per 

.282 .058 < p < .482 
.174 -.005 < P < .265 

.721 .022 < p <.442 

•190 -.050 < P < .178 

.176 .100 < p < .524 

.569 .130 < p < .463 

^e shrinkage formula provided by Cohen & Cohen (1983) was applied 
to multiple R's from individual studies with multiple objective 
indices predicting a singular subjective rating of overall 
effectiveness. 



Black .108 .0294 

Overall 

White .069 .0226 

Performance Black <107 QQ ^ 
Indices 

White .058 .0145 

Job Knowledge Black .108 .0542 
Indices 

White .085 .0055 
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